Papers with attention mechanism
Adaptive Transformers for Learning Multimodal Representations (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches for learning visiolinguistic representations with transformers are over-parametrized and require extensive training. |
| Approach: | They propose to extend attention spans, sparse, and structured dropout methods to learn more about how the network perceives the complexity of input sequences. |
| Outcome: | The proposed approaches improve on language semantics and visiolinguistic representations, but are often over-parametrized and require large amounts of computation. |
Systematicity Emerges in Transformers when Abstract Grammatical Roles Guide Attention (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing systems that use transformers lack systematicity, but they are inferior to human learners in sample efficiency and difficult generalization problems. |
| Approach: | They propose to modify a transformer so that it controls attention distributions and fills in the gaps. |
| Outcome: | The proposed model shows that the performance of natural language processing systems is improved when abstract role labels are assigned to the input stream and provided to the role stream. |
A Cluster Ranking Model for Full Anaphora Resolution (2020.lrec-1)
Copied to clipboard
| Challenge: | Anaphora resolution systems designed for CONLL 2012 dataset can handle key aspects of the full anaphora task such as the identification of singletons and of certain types of non-referring expressions. |
| Approach: | They propose an architecture to identify non-referring expressions and build coreference chains, including singletons, using system mentions. |
| Outcome: | The proposed model performs better on the CONLL 2012 dataset than the state-of-the-art system. |
Automatic Rule Induction for Efficient Semi-Supervised Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to generalize from labeled and unlabeled data are difficult to explain and behave unreliably. |
| Approach: | They propose a framework for automatic discovery and integration of symbolic rules into pretrained transformer models by using an attention mechanism. |
| Outcome: | The proposed framework can improve state-of-the-art methods with no manual effort and minimal computational overhead. |
Transformer-Based Direct Hidden Markov Model for Machine Translation (2021.acl-srw)
Copied to clipboard
| Challenge: | Recent studies have found that word alignments produced by the multi-head cross-attention weights are poor. |
| Approach: | They propose to introduce the hidden Markov model to the transformer architecture and introduce alignment components while keeping the system monolithic. |
| Outcome: | The proposed model outperforms the baseline model but is slower in training and decoding. |
Improving Conversational Recommendation Systems’ Quality with Context-Aware Item Meta-Information (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to integrate the recommendation function and dialog generation function smoothly are lacking. |
| Approach: | They propose to integrate dialog context for recommendation and dialog generation better using a pre-trained language model and an item metadata encoder to integrate the recommendation and dialogue generation. |
| Outcome: | The proposed architecture improves the integration of recommendation and dialog generation functions. |
Don’t Shoot The Breeze: Topic Continuity Model Using Nonlinear Naive Bayes With Attention (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) are becoming increasingly popular in business scenarios, but maintaining topic continuity is a challenge. |
| Approach: | They propose a topic continuity model that assesses whether a response aligns with the initial conversation topic using a Naive Bayes approach. |
| Outcome: | The proposed model outperforms existing models in handling lengthy and complex conversations. |
Deep Bayesian Natural Language Processing (P19-4)
Copied to clipboard
| Challenge: | Introduction to deep Bayesian learning for natural language addresses the fundamentals of statistical models and neural networks. |
| Approach: | This tutorial addresses the advances in deep Bayesian learning for natural language . it focuses on advanced Bayessian models and deep models . authors present case studies and domain applications to tackle different issues . |
| Outcome: | This tutorial focuses on advanced Bayesian models and deep models for natural language . case studies and domain applications are presented to tackle different issues in deep Bayessian processing, learning and understanding. |
A Multiscale Visualization of Attention in the Transformer Model (P19-3)
Copied to clipboard
| Challenge: | Various tools have been developed to visualize attention in NLP models, ranging from attention-matrix heatmaps to bipartite graph representations. |
| Approach: | They propose an open-source tool that visualizes attention at multiple scales and provides a unique perspective on the attention mechanism. |
| Outcome: | The proposed model outperforms OpenAI GPT-2 and BERT on several language modeling benchmarks. |
Acceptability Judgements via Examining the Topology of Attention Maps (2022.findings-emnlp)
Copied to clipboard
Daniil Cherniavskii, Eduard Tulchinskii, Vladislav Mikhailov, Irina Proskurina, Laida Kushnareva, Ekaterina Artemova, Serguei Barannikov, Irina Piontkovskaya, Dmitri Piontkovski, Evgeny Burnaev
| Challenge: | Acceptability judgments are a key component of generative linguistics, but their ability to judge grammatical acceptability has not been explored. |
| Approach: | They propose to exploit the geometric properties of the attention graph to evaluate the grammatical acceptability of sentences using topological data analysis. |
| Outcome: | The proposed approach outperforms nine statistical and Transformer LM baselines on the BLiMP benchmark and the human-level performance on the same benchmark. |
Thesis Proposal: Efficient KV Cache Reuse for Multi-Document Retrieval-Augmented Generation (2026.eacl-srw)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) systems face efficiency bottlenecks in prefill due to attention mechanism, and traditional KV cache only accelerates decoding. |
| Approach: | They propose a multi-document KV cache reuse framework for multi-doc RAG workloads . they propose to resolve position and context misalignment while eliminating document-specific quadratic complexity in prefill. |
| Outcome: | The proposed framework solves position and context misalignment issues while eliminating document-specific quadratic complexity in prefill. |
Going “Deeper”: Structured Sememe Prediction via Transformer with Tree Attention (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies ignore hierarchical structures of sememes in sememe-based semantic description systems. |
| Approach: | They propose a structured sememe prediction problem to predict a sememes tree with hierarchical structures rather than a set of sememas. |
| Outcome: | The proposed model outperforms baseline models and shows its effectiveness . it predicts a sememe tree with hierarchical structures rather than a set of sememes . |
Entity-Consistent End-to-end Task-Oriented Dialogue System with KB Retriever (D19-1)
Copied to clipboard
| Challenge: | Existing work on sequence-to-sequence dialogues treats the KB query as an attention over the entire KB without the guarantee that the generated entities are consistent with each other. |
| Approach: | They propose a framework which queries the knowledge base in two steps to improve consistency . they first return the most relevant KB row given a dialogue history . |
| Outcome: | The proposed framework outperforms baseline models and produces entity-consistent responses. |
Exploiting WordNet Synset and Hypernym Representations for Answer Selection (2020.aacl-main)
Copied to clipboard
| Challenge: | Answer selection (AS) is a challenging subtask of document-based question answering (DQA). |
| Approach: | They propose to use WordNet to enrich the word representation and sentence encoding to incorporate similarity scores of two concepts that share synset or hypernym relations into the attention mechanism. |
| Outcome: | The proposed model outperforms existing state-of-the-art models on the public WikiQA and SelQA datasets and significantly improves the baseline system. |
Dodrio: Exploring Transformer Models with Interactive Visualization (2021.acl-demo)
Copied to clipboard
| Challenge: | Recent research suggests the key may lie in multi-headed attention mechanism’s ability to learn and represent linguistic information. |
| Approach: | They present an open-source visualization tool to analyze attention mechanisms in transformer-based models with linguistic knowledge. |
| Outcome: | Dodrio analyzes attention mechanisms in transformer-based models with linguistic knowledge. |
CELI: Simple yet Effective Approach to Enhance Out-of-Domain Generalization of Cross-Encoders. (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing cross-encoders do not capture all information into the [CLS] token . Xiong et al., 2021) find that the out-of-domain approach is less robust. |
| Approach: | They introduce a cross-encoder with late interaction that incorporates a late interaction layer into existing models. |
| Outcome: | The proposed method improves BEIR by 5% without compromising in-domain effectiveness or search latency. |
HiPool: Modeling Long Documents Using Graph Neural Networks (2023.acl-short)
Copied to clipboard
| Challenge: | Recent work on pretraining languages have achieved satisfying results in many NLP tasks, but they are still restricted by a pre-defined maximum length. |
| Approach: | They propose a graph-based method to model sentence-level information using a fixed length and graphs to model intra- and cross-sentence correlations. |
| Outcome: | The proposed model outperforms baseline models by 2.6% in F1 score, and 4.8% on the longest sequence dataset. |
Natural Answer Generation with Heterogeneous Memory (N18-1)
Copied to clipboard
| Challenge: | Recent work on memory augmented encoder-decoder frameworks has shown promising progress for natural language generation tasks. |
| Approach: | They propose a memory-augmented encoder-decoder framework that takes care of memory contents from different sources to explicitly avoid repetition. |
| Outcome: | The proposed approach can produce readable and meaningful answer sentences while maintaining high coverage for given answer information. |
Measuring Alignment Bias in Neural Seq2seq Semantic Parsers (2022.starsem-1)
Copied to clipboard
| Challenge: | Sequence-to-sequence semantic parsers with attention mechanisms have changed the research landscape . emergence of seq2seq models have led to questions about alignments . |
| Approach: | They investigate whether seq2seq models can handle both simple and complex alignments. |
| Outcome: | The proposed model performs better on monotonic and complex alignments compared to monotonic models . |
Dilated LSTM with attention for Classification of Suicide Notes (D19-62)
Copied to clipboard
| Challenge: | Using a dilated LSTM with attention we achieve an accuracy of 87.34% compared to baselines of 80.35% and 82.27%. |
| Approach: | They propose a dilated LSTM with attention mechanism for document-level classification of suicide notes, last statements and depressed notes. |
| Outcome: | The proposed model achieves an accuracy of 87.34% compared to baselines of 80.35% and 82.27%. |
HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding (2025.coling-main)
Copied to clipboard
| Challenge: | Object categories are typically organized into a multi-granularity taxonomic hierarchy . traditional uni-modal approaches focus primarily on image features, revealing limitations in complex scenarios. |
| Approach: | They propose a framework that combines vision-language models with a deeper exploitation of the hierarchy. |
| Outcome: | The proposed framework shows significant improvements on 11 diverse visual recognition benchmarks. |
Muted: Multilingual Targeted Offensive Speech Identification and Visualization (2023.emnlp-demo)
Copied to clipboard
Christoph Tillmann, Aashka Trivedi, Sara Rosenthal, Santosh Borse, Rong Zhang, Avirup Sil, Bishwaranjan Bhattacharjee
| Challenge: | Existing visualizations of offensive language use only sentence level annotations, but there are few that explore spans and other languages. |
| Approach: | They propose a system to identify multilingual HAP content by displaying offensive arguments and their targets using heat maps to indicate their intensity. |
| Outcome: | The proposed model can identify toxic spans without further fine-tuning using existing models and its attention mechanism out-of-the-box. |
Big Bidirectional Insertion Representations for Documents (D19-56)
Copied to clipboard
| Challenge: | Recent studies suggest we are nearing human-level parity for sentence-level translation in certain domains. |
| Approach: | They propose an insertion-based model for document-level translation tasks that embeds sentence alignment between the source and target document. |
| Outcome: | The proposed model improves on the WMT’19 English->German translation task by +4.3 BLEU compared with the Insertion Transformer baseline. |
How does Attention Affect the Model? (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the effectiveness of attention in NLP do not consider changes in semantic capability of different components. |
| Approach: | They propose a framework that exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines indicators to capture the impact of attention on sequence semantic. |
| Outcome: | The proposed framework exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines indicators to capture the impact of attention on sequence semantic. |
A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to contrastive learning are heavily affected by superficial features like sentence length and syntax. |
| Approach: | They propose a semantic-aware contrastive learning framework for sentence embeddings that explores the pseudo-token space representation of a sentence while eliminating the impact of superficial features such as sentence length and syntax. |
| Outcome: | The proposed framework outperforms the state-of-the-art on six standard semantic textual similarity tasks while maintaining an additional queue to store the representation of sentence embeddings. |
PR-XAI: PageRank-Based Feature Attribution for Transformers (2026.acl-long)
Copied to clipboard
| Challenge: | Existing feature attribution methods for transformer models suffer from limitations that undermine their efficacy. |
| Approach: | They propose a feature attribution method for transformer models based on PageRank . they propose attribution methods that apply PageRank to attention-derived graphs . |
| Outcome: | The proposed method outperforms state-of-the-art methods in faithfulness and classification metrics with significant gains on long-form text. |
MINER: Multi-Interest Matching Network for News Recommendation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods learn a single user embedding from user’s historical behaviors to represent the reading interest. |
| Approach: | They propose a poly attention scheme to learn multiple interest vectors for each user, which encodes the different aspects of user interest. |
| Outcome: | The proposed approach significantly outperforms existing state-of-the-art methods on the MIND news recommendation benchmark. |
Learning beyond Datasets: Knowledge Graph Augmented Neural Networks for Natural Language Processing (N18-1)
Copied to clipboard
| Challenge: | Currently, machine learning is limited in scalability and is limited to specific training data. |
| Approach: | They propose to enhance learning models with world knowledge in the form of Knowledge Graph fact triples for natural language processing tasks. |
| Outcome: | The proposed method is highly scalable to the amount of prior information that has to be processed and can be applied to any generic NLP task. |
Weakly Supervised Attentional Model for Low Resource Ad-hoc Cross-lingual Information Retrieval (D19-61)
Copied to clipboard
| Challenge: | Low resource languages often lack relevance annotations for cross-lingual information retrieval . when available, the training data has limited coverage for possible queries . |
| Approach: | They propose a weakly supervised neural model for Cross-lingual information retrieval from low-resource languages using weak supervision instead of relevance annotations. |
| Outcome: | The proposed model achieves 19 MAP points improvement compared to CNNs and 12 points improvement from machine translation-based CLIR models. |
Neural OCR Post-Hoc Correction of Historical Corpora (2021.tacl-1)
Copied to clipboard
| Challenge: | Optical character recognition (OCR) is crucial for a deeper access to historical collections. |
| Approach: | They propose a neural approach based on a combination of recurrent (RNN) and deep convolutional network (ConvNet) to correct OCR transcription errors. |
| Outcome: | The proposed model reduces the word error rate of 32.3% by more than 89% on a historical book corpus in German language. |
Cross-layer Attention Sharing for Pre-trained Large Language Models (2026.tacl-1)
Copied to clipboard
Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Jiali Zeng, Qiaozhi He, Murun Yang, Fandong Meng, Jie Zhou, Tong Xiao, Jingbo Zhu
| Challenge: | Existing studies focus on compressing the Key-Value cache or grouping attention heads, while overlooking redundancy between layers. |
| Approach: | They propose a lightweight substitute for self-attention in well-trained LLMs that uses feed-forward networks to align attention heads between adjacent layers and low-rank matrices to approximate differences in layer-wise attention weights. |
| Outcome: | The proposed model reduces redundancy by sharing weights across layers while maintaining high response quality while reducing redundant calculations within 53% 84% of the total layers. |
Attention over Heads: A Multi-Hop Attention for Neural Machine Translation (P19-2)
Copied to clipboard
| Challenge: | Existing multihop attentions for machine comprehension are recurrent and hierarchical . a proposed multi-hop attention for the Transformer refines the attention for an output symbol many times . |
| Approach: | They propose a multi-hop attention for the Transformer which integrates attentions from each head. |
| Outcome: | The proposed model outperforms the baseline Transformer in terms of translation accuracy and speed. |
Extending Event Detection to New Types with Learning from Keywords (D19-55)
Copied to clipboard
| Challenge: | Existing methods for event detection classify words or phrases into specific types of interest. |
| Approach: | They propose a new event detection formulation that describes types via keywords to match contexts in documents. |
| Outcome: | The proposed formulation improves the performance of the proposed model to new types. |
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies suggest multimodal transformer models learn rich visual-linguistic representations. |
| Approach: | They focus on dataset noise and language similarity to their downstream task . they find that models with a multimodal attention mechanism outperform deeper models with modality-specific attention mechanisms. |
| Outcome: | The proposed models outperform models with a multimodal attention mechanism on downstream tasks. |
Attention and Lexicon Regularized LSTM for Aspect-based Sentiment Analysis (P19-2)
Copied to clipboard
| Challenge: | End-to-end deep learning systems lack flexibility as one cannot adjust the network to fix an obvious problem. |
| Approach: | They propose a way to leverage lexicon information to make the model more flexible . they also explore the effect of regularizing attention vectors to allow the network to have a broader "focus" |
| Outcome: | The proposed approach leverages lexicon information to make it more flexible and robust. |
Synchronous Syntactic Attention for Transformer Neural Machine Translation (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing syntaxbased NMT models use monolingual syntactic information on either side or both. |
| Approach: | They propose a mechanism that synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target- and target side self- attentions mapped by the encoder-decoder attention matrix. |
| Outcome: | The proposed method improves translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU). |
C2DLM: Causal Concept-Guided Diffusion Large Language Models (2026.findings-acl)
Copied to clipboard
Kairong Han, Nuanqiao Shan, Ziyu Zhao, Zijing Hu, Xinpeng Dong, Ye Jun Jian, Lujia Pan, Fei Wu, Kun Kuang
| Challenge: | Autoregressive (AR) and diffusion language models (DLMs) suffer from insufficient reasoning capabilities. |
| Approach: | They propose a fully connected Diffusion Language Model that uses a concept-level causal graph to guide attention to learn causal relationships between concepts. |
| Outcome: | The proposed model achieves a 12% improvement and 3.2 training speedup on the COT-OrderPerturb task, along with an average gain of 1.31% across six downstream reasoning tasks. |
Accurate Word Alignment Induction from Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prior work suggests that Transformer captures poor word alignments through its attention mechanism. |
| Approach: | They propose two new word alignment induction methods that use attention weights to capture accurate word alignments. |
| Outcome: | The proposed methods outperform baselines on three publicly available datasets and are significantly better than GIZA++. |
Enriching Neural Models with Targeted Features for Dementia Detection (P19-2)
Copied to clipboard
| Challenge: | In the United States, adults over 65 are expected to comprise one-fifth of the population by 2030, and a larger proportion of the . population than those under 18 by 2035. |
| Approach: | They propose a neural model that takes into account both long language samples and hand-crafted linguistic features to distinguish between dementia affected and healthy patients. |
| Outcome: | The proposed model achieves an F1 score of 0.929 on the DementiaBank dataset and the state-of-the-art on the dataset. |
Hierarchical Attention Prototypical Networks for Few-Shot Text Classification (D19-1)
Copied to clipboard
| Challenge: | Existing methods for text classification are based on large-scale labeled data, but few data are available. |
| Approach: | They propose a hierarchical attention prototypical networks for few-shot text classification . they use attention mechanism to highlight or weaken the importance of features, words, and instances . |
| Outcome: | The proposed model can capture more important features, words, and instances . it can also increase support set augmentability and accelerate convergence speed in training stage . |
De-Mixing Sentiment from Code-Mixed Text (P19-2)
Copied to clipboard
| Challenge: | Code-mixing is the phenomenon of mixing the vocabulary and syntax of multiple languages in the same sentence. |
| Approach: | They propose a hybrid architecture for the task of Sentiment Analysis of English-Hindi code-mixed data using CNNs to generate subword representations for the sentences. |
| Outcome: | The proposed architecture achieves 83.54% accuracy and 0.827 F1 score on a benchmark dataset. |
Reversed Attention: On The Gradient Descent Of Attention Layers In GPT (2025.naacl-long)
Copied to clipboard
| Challenge: | In this work, we examine the attention maps obtained from the backward pass of attention, which we call "Reversed Attention" (RA). |
| Approach: | They propose to use a method called "attention patching" to alter the forward pass of attention without modifying the model's weights. |
| Outcome: | The proposed method enables the model to alter the forward pass of attention without altering the model’s weights. |
LongT5: Efficient Text-To-Text Transformer for Long Sequences (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that increasing the input length or increasing model size can improve the performance of Transformer-based neural models. |
| Approach: | They propose a model that integrates attention ideas from long-input transformers and adopts pre-training strategies from summarization pre-train into the scalable T5 architecture. |
| Outcome: | The proposed model outperforms the original T5 models on several summarization and question answering tasks and achieves state-of-the-art results. |
Spoken Language Understanding for Task-oriented Dialogue Systems with Augmented Memory Networks (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent research shows promising results by jointly learning of slot filling and intent detection tasks. |
| Approach: | They propose a way to combine slot filling and slot filler learning to achieve state-of-the-art results. |
| Outcome: | The proposed model outperforms existing methods on benchmark datasets and ATIS datasets. |
Entity Commonsense Representation for Neural Abstractive Summarization (N18-1)
Copied to clipboard
| Challenge: | Current ELS’s are not sufficiently effective, possibly introducing unresolved ambiguities and irrelevant entities. |
| Approach: | They propose an off-the-shelf entity linking system to extract linked entities and propose Entity2Topic (E2T) module attachable to a sequence-to-sequence model that transforms a list of entities into a vector representation of the topic of the summary. |
| Outcome: | The proposed model improves the performance of the Gigaword and CNN summarization datasets by at least 2 ROUGE points. |
Surprisingly Easy Hard-Attention for Sequence to Sequence Learning (D18-1)
Copied to clipboard
| Challenge: | Existing attention mechanisms are hard and hard, but they are more accurate when trained. |
| Approach: | They propose to use a beam approximation of the joint distribution between attention and output to train sequence to sequence learning. |
| Outcome: | The proposed method is compared to existing attention mechanisms on five translation tasks and shows consistent gains on the same tasks. |
Dynamic Feature Selection with Attention in Incremental Parsing (C18-1)
Copied to clipboard
| Challenge: | Currently, incremental transition-based parsers require that all inputs are visible from the beginning to extract good features from a limited local context. |
| Approach: | They propose a technique to maximize local features with an attention mechanism which works as context- dependent dynamic feature selection. |
| Outcome: | The proposed technique can extract features from a limited local context and is able to perform multilingual experiments and demon strate on local ambiguous points. |
Robust Dialogue State Tracking with Weak Supervision and Sparse Data (2022.tacl-1)
Copied to clipboard
Michael Heck, Nurul Lubis, Carel van Niekerk, Shutong Feng, Christian Geishauser, Hsien-Chin Lin, Milica Gašić
| Challenge: | Generalizing dialogue state tracking (DST) to new data and domains is especially challenging due to the strong reliance on abundant and fine-grained supervision during training. |
| Approach: | They propose a training strategy to build extractive DST models without the need for fine-grained manual span labels. |
| Outcome: | The proposed model improves robustness against sample sparsity, new concepts, and topics, leading to state-of-the-art performance on a range of benchmarks. |
Rigid Formats Controlled Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Neural text generation is a challenging task that requires rigid formats to be controlled . a framework called SongNet is designed to tackle this problem . |
| Approach: | They propose a framework to tackle a task called rigid formats controlled text generation . they propose rhyming schemes and a transformer-based auto-regressive language model to improve the modeling performance . |
| Outcome: | The proposed framework improves the performance on format, rhyme, and sentence integrity. |
Attention Transfer Network for Aspect-level Sentiment Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Aspect-level sentiment classification aims to detect the sentiment polarity of a given opinion target in a sentence. |
| Approach: | They propose a novel attention transfer network which can exploit attention from document-level sentiment datasets to improve the attention capability of the aspect-level classification task. |
| Outcome: | The proposed method outperforms state-of-the-art methods on two ASC benchmark datasets. |
Self-Attention with Relative Position Representations (N18-2)
Copied to clipboard
| Challenge: | Recent approaches to sequence to sequence learning leverage recurrence, convolution, attention or combination of recurrent and convolutional neural networks. |
| Approach: | They propose an approach that extends the self-attention mechanism to consider representations of relative positions, or distances between sequence elements. |
| Outcome: | The proposed approach yields 1.3 BLEU and 0.3 BLUE on translation tasks . it is based on a relation-aware self-attention mechanism that can generalize to arbitrary graph-labeled inputs. |
Modeling Code-Switch Languages Using Bilingual Parallel Corpus (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for bilingual language modeling are limited due to lack of training data and syntactic structure. |
| Approach: | They propose a bilingual attention language model that performs language modeling objective with a quasi-translation objective to model the monolingual and cross-lingual sequential dependency. |
| Outcome: | The proposed model reduces the perplexity of 20.5% over the best-reported model. |
Transformation Networks for Target-Oriented Sentiment Classification (P18-1)
Copied to clipboard
| Challenge: | a new model for sentiment classification uses attention instead of attention to classify sentiment polarities over individual opinion targets. |
| Approach: | They propose a model that uses a CNN layer to extract salient features from transformed word representations from a bi-directional RNN layer. |
| Outcome: | The proposed model achieves state-of-the-art on a few benchmarks. |
Target-Sensitive Memory Networks for Aspect Sentiment Classification (P18-1)
Copied to clipboard
| Challenge: | Aspect sentiment classification (ASC) is a fundamental task in sentiment analysis. |
| Approach: | They propose to use memory networks to deal with ASC using aspect and sentence terms and use them to classify the sentiment polarity. |
| Outcome: | The proposed techniques can be implemented in a variety of contexts and their effectiveness is evaluated. |
DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks (2022.findings-naacl)
Copied to clipboard
| Challenge: | Experimental results show that Transformer Encoder model can't automatically capture word order, so explicit position embeddings are required to be fed into the target model. |
| Approach: | They propose a Transformer-based language model DecBERT that uses a causal attention mask to capture word order. |
| Outcome: | The proposed model improves on the GLUE language understanding benchmark and accelerates the pre-training process. |
Flow-Adapter Architecture for Unsupervised Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in deep learning have boosted the development of neural machine translation (NMT). |
| Approach: | They propose a flow-adapter architecture for unsupervised neural machine translation that leverages normalizing flows to model distributions of sentence-level latent representations. |
| Outcome: | The proposed model achieves competitive results on several unsupervised MT benchmarks. |
Effective Attention Modeling for Aspect-Level Sentiment Classification (C18-1)
Copied to clipboard
| Challenge: | Aspect-level sentiment classification aims to determine sentiment polarity of review sentence towards opinion target . main challenge is to separate different opinion contexts for different targets . |
| Approach: | They propose a method that captures the semantic meaning of the opinion target and a model that incorporates syntactic information into the attention mechanism. |
| Outcome: | The proposed method captures the semantic meaning of the opinion target and incorporates syntactic information into the attention mechanism. |
Don’t Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Vision Language Models suffer from hallucinations, attributing incorrect or misleading features to images. |
| Approach: | They propose a test-time approach that recalibrates the influence of blind tokens . they identify blind token by analyzing layer-wise attention distributions over image tokens. |
| Outcome: | The proposed approach reduces hallucinations in large vision language models . it uses a contrastive decoding strategy to balance the influence of blind tokens . |
Explainable Prediction of Medical Codes from Clinical Text (N18-1)
Copied to clipboard
| Challenge: | Clinical notes are text documents that are created by clinicians for each patient encounter. |
| Approach: | They propose a method that aggregates information across the document using a convolutional neural network and uses an attention mechanism to select the most relevant segments for each of the thousands of possible codes. |
| Outcome: | The proposed method is accurate and better than the current state of the art. |
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models use the attention mechanism to contextualize input inputs . but, we find that it is not as important as thought for pretrained models . |
| Approach: | They propose a probing method that replaces input-dependent attention matrices with constant ones. |
| Outcome: | The proposed method improves performance of pretrained language models without input-dependent attention. |
Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network (P18-1)
Copied to clipboard
| Challenge: | Existing models for matching dialogue responses rely on semantic and functional dependencies . a recent study only uses the last utterance in context for matching a reply . |
| Approach: | They propose a model that matches a response with its multi-turn context using attention. |
| Outcome: | The proposed model outperforms the state-of-the-art models on two large-scale multi-turn response selection tasks. |
Supervised Domain Enablement Attention for Personalized Domain Classification (D18-1)
Copied to clipboard
| Challenge: | Recent IPDAs cover more than several thousands of diverse domains including Alexa Skills, Google Actions, and Cortana Skills. |
| Approach: | They propose a supervised enablement attention mechanism that utilizes sigmoid activation for the attention weighting and self-distillation to leverage the attention information of other enabled domains. |
| Outcome: | The proposed approach improves domain classification performance on real-world domains. |
On-the-Fly Attention Modulation for Neural Generation (2021.findings-acl)
Copied to clipboard
Yue Dong, Chandra Bhagavatula, Ximing Lu, Jena D. Hwang, Antoine Bosselut, Jackie Chi Kit Cheung, Yejin Choi
| Challenge: | Degeneration of neural text is associated with insufficient learning of task-specific characteristics by the attention mechanism. |
| Approach: | They propose to use attention modulation to inject priors into inference to improve fluency, creativity, and commonsense reasoning in neural text generation models. |
| Outcome: | The proposed method improves fluency, creativity, and commonsense reasoning, and significantly reduces sentence-level repetition. |
Higher-Order Coreference Resolution with Coarse-to-Fine Inference (N18-2)
Copied to clipboard
| Challenge: | a new approach to coreference resolution uses a span-ranking architecture as an attention mechanism to iteratively refine span representations. |
| Approach: | They propose a fully-differentiable approximation to higher-order inference for coreference resolution . they propose introducing a coarse-to-fine approach that incorporates a less accurate but more efficient bilinear factor . |
| Outcome: | The proposed model significantly improves accuracy on the English OntoNotes benchmark while being far more computationally efficient. |
Rumor Detection by Exploiting User Credibility Information, Attention and Multi-task Learning (P19-1)
Copied to clipboard
| Challenge: | Social media platforms do not always pose authentic information, and rumors spread fear or hate. |
| Approach: | They propose a new multi-task learning approach for rumor detection and stance classification tasks. |
| Outcome: | The proposed model outperforms the state-of-the-art rumor detection approaches on two datasets. |
Event Detection with Neural Networks: A Rigorous Empirical Evaluation (D18-1)
Copied to clipboard
| Challenge: | Neural network models have been the most successful for event detection, but they ignore syntactic relationships in the text. |
| Approach: | They propose a GRU-based model that combines syntactic information along with temporal structure through an attention mechanism. |
| Outcome: | The proposed model is competitive with existing models on a ACE2005 dataset. |
On the Word Alignment from Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Prior researches suggest that neural machine translation (NMT) captures word alignment through its attention mechanism, however, attention may fail to capture word alignment for some NMT models. |
| Approach: | They propose two methods to induce word alignment which are general and agnostic to specific NMT models. |
| Outcome: | The proposed methods induce much better word alignment than attention. |
Self-Attentive Residual Decoder for Neural Machine Translation (N18-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence networks with attention have been used for machine translation . however, the target-side context is limited and the model lacks the ability to capture non-syntactic dependencies among words. |
| Approach: | They propose a sequence-to-sequence network with attention that captures contextual information at each time-step prediction through an attention mechanism. |
| Outcome: | The proposed model outperforms a neural MT baseline and memory and self-attention network on three language pairs. |
Are BLEU and Meaning Representation in Opposition? (P18-1)
Copied to clipboard
| Challenge: | Empirical evaluation suggests that the better the translation quality, the worse the learned sentence representations serve in a wide range of classification and similarity tasks. |
| Approach: | They propose several variations of the attentive NMT architecture to bring this meeting point back . they propose to use a structured fixed-size representation of the input to produce static representations of input sentences. |
| Outcome: | The proposed architecture improves translation quality and performance in a range of tasks. |
DoLFIn: Distributions over Latent Features for Interpretability (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to interpret neural networks face a trade-off between a model's usefulness and its complexity. |
| Approach: | They propose a novel approach to achieve interpretability that avoids this trade-off by using probability as the central quantity instead of a fixed quantity. |
| Outcome: | The proposed approach outperforms the classical CNN and BiLSTM classifiers on the SST2 and AG-news datasets. |
Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together (N19-1)
Copied to clipboard
| Challenge: | Neural networks equipped with self-attention have parallelizable computation and the ability to capture both long-range and local dependencies. |
| Approach: | They propose a novel attention mechanism called "Multi-mask Tensorized Self-Attention" it captures pairwise and global dependencies by a compatibility function composed of dot-product and additive attentions . |
| Outcome: | The proposed model outperforms CNN-/RNN-/attention-based models on nine NLP benchmarks with compelling memory- and time-efficiency. |
Adaptive Attentional Network for Few-Shot Knowledge Graph Completion (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent attempts to learn static representations of entities and references ignore their dynamic properties. |
| Approach: | They propose to learn static representations of entities and references ignoring their dynamic properties . a neighbor encoder learns entities' roles while a query-aware aggregator learns references' contributions . |
| Outcome: | The proposed approach achieves state-of-the-art results with different few-shot sizes. |
Rethinking Long Context Generation from the Continual Learning Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with processing long contexts due to the limited context window. |
| Approach: | They propose to combine a limited context window with a continual learning perspective to improve LLMs' efficiency in processing long contexts. |
| Outcome: | The proposed models improve the performance of Large Language Models (LLMs) by integrating learning strategies with existing approaches. |
Few-shot Learning for Slot Tagging with Attentive Relational Network (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have used metric-based learning in computer vision but not slot tagging. |
| Approach: | They propose a metric-based learning architecture that extends relation networks by leveraging pretrained contextual embeddings such as ELMO and BERT and by using attention mechanism. |
| Outcome: | The proposed method outperforms state-of-the-art methods on SNIPS data on a slot tagging task with a large amount of hand-labeled data. |
Sparse Sequence-to-Sequence Models (P19-1)
Copied to clipboard
| Challenge: | Sequence-to-sequence models are dense and assigning nonzero probability to implausible outputs. |
| Approach: | They propose a new family of -entmax transformations that includes softmax and sparsemax as particular cases and is sparser for any > 1 . they provide fast algorithms to evaluate these transformations and their gradients, which scale well for large vocabulary sizes. |
| Outcome: | The proposed models are able to produce sparse alignments and assign nonzero probability to short list of plausible outputs, sometimes rendering beam search exact. |
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Long-context modeling has drawn more attention in the area of Large Language Models (LLMs). |
| Approach: | They propose a Long-context data selection framework with Attention-based Dependency Measurement which can efficiently identify high-quality long-contrast data from a large-scale, multi-domain pre-training corpus. |
| Outcome: | The proposed framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training. |
Grounded Graph Decoding improves Compositional Generalization in Question Answering (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Current compositional generalization models lose syntax context when learning a flat input . a new method to improve compositional globalization is proposed to ground structured predictions with an attention mechanism. |
| Approach: | They propose a method to ground structured predictions by a structure-based attention mechanism. |
| Outcome: | The proposed method performs competitively on the Compositional Freebase Questions dataset. |
Visual Question Answering Dataset for Bilingual Image Understanding: A Study of Cross-Lingual Transfer Using Attention Maps (C18-1)
Copied to clipboard
| Challenge: | Existing literature on visual question answering (VQA) focuses on English, but there are no datasets for other languages. |
| Approach: | They propose a cross-lingual method to make use of English annotation to improve Japanese VQA . they use attention maps generated from English questions to improve the task . |
| Outcome: | The proposed method performs better than using a monolingual corpus in Japanese than using monolingual ones. |
Grouped-Attention for Content-Selection and Content-Plan Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent neural data-to-text generation models explicitly learn content-plan given a set of attributes as input. |
| Approach: | They propose a neural content-planner that captures local and global contexts . they use a token-level attention constrained within each input attribute . |
| Outcome: | The proposed model outperforms competitors by 4.92%, 4.70%, and 16.56% on real-world datasets. |
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer (2022.acl-long)
Copied to clipboard
| Challenge: | Existing sparse attention methods use fixed patterns to select words without considering similarities between words. |
| Approach: | They propose a neural clustering method which integrates into the Self-Attention Mechanism in Transformer and integrates it into the target task. |
| Outcome: | The proposed method outperforms two typical sparse attention methods on translation, text classification, and text matching tasks while having a comparable or even better time and memory efficiency. |
Leveraging Gloss Knowledge in Neural Word Sense Disambiguation by Hierarchical Co-Attention (D18-1)
Copied to clipboard
| Challenge: | Existing models for Word Sense Disambiguation use labeled data, but lack gloss knowledge. |
| Approach: | They propose a co-attention mechanism to generate co-dependent representations for context and gloss . they propose to incorporate gloss knowledge into neural networks for Word Sense Disambiguation . |
| Outcome: | The proposed model achieves state-of-the-art results on standard English all-words WSD datasets. |
Syntax-guided Localized Self-attention by Constituency Syntactic Distance (2022.findings-emnlp)
Copied to clipboard
Shengyuan Hou, Jushi Kai, Haotian Xue, Bingyu Zhu, Bo Yuan, Longtao Huang, Xinbing Wang, Zhouhan Lin
| Challenge: | Recent studies have shown that Transformers is implicitly learning syntactic information from data, albeit is highly dependent on the quality and scale of the training data. |
| Approach: | They propose a syntax-guided localized self-attention model that allows directly incorporating grammar structures from an external constituency parser. |
| Outcome: | The proposed model improves translation performance on a variety of datasets, from small to large datasets and with different source languages. |
Adaptive Learning of Local Semantic and Global Structure Representations for Text Classification (C18-1)
Copied to clipboard
Jianyu Zhao, Zhiqiang Zhan, Qichuan Yang, Yang Zhang, Changjian Hu, Zhensheng Li, Liuxin Zhang, Zhiqiang He
| Challenge: | Existing representation models for text classification learn little structure information or rely on pre-defined structures. |
| Approach: | They propose a sandwich neural network to learn local semantic and global structure representations without relying on parsers. |
| Outcome: | The proposed approach achieves competitive performance on several text classification tasks. |
Attention-based Contextual Language Model Adaptation for Speech Recognition (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing language models do not incorporate utterance level contextual information . however, for some domains like voice assistants, additional context provides a rich input signal . |
| Approach: | They propose a method for training neural speech recognition models on text and contextual data. |
| Outcome: | The proposed model reduces perplexity by 7.0% relative over a standard LM . it also improves perxicity by 2.8% relative to a state-of-the-art model for contextual LM. |
Label-Driven Denoising Framework for Multi-Label Few-Shot Aspect Category Detection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for ACD use label information of aspect categories to detect aspect categories . but, they still suffer from noise problems due to lack of supervised data . |
| Approach: | They propose a Label-Driven Denoising Framework to alleviate noise problems for ACD subtask . they use the label information of each aspect to generate a better prototype . |
| Outcome: | The proposed framework improves the performance of the multi-label few-shot Aspect Category Detection task. |
A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classification (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing deep learning models have the attention mechanism to improve performance, but the inherent characteristics of deep learning model complexity and the flexibility of the attention structure make them difficult to explain. |
| Approach: | They propose a two-tier attention architecture to decouple the complexity of explanation and the decision-making process by using large-scale news corpora. |
| Outcome: | The proposed model can achieve competitive performance with state-of-the-art models and illustrates its appropriateness from an explainability perspective. |
DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing models that take text block positions into account are not efficient for document understanding. |
| Approach: | They propose a layout-aware BERT model that takes into account text block positions in relative polar coordinate system rather than the Cartesian one. |
| Outcome: | The proposed model eliminates the need for absolute positional embeddings on a dataset more than six times smaller than the widely used IIT-CDIP corpus. |
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained transformer models have demonstrated remarkable performance across various natural language processing tasks. |
| Approach: | They propose a transformer variant with mixed attention spans that leverages the attention mechanism to capture long- and short-range dependencies in the sequence. |
| Outcome: | The proposed model can achieve competitive performance to models with full attention while reducing computational cost (75%) |
Combining Distant and Direct Supervision for Neural Relation Extraction (N19-1)
Copied to clipboard
| Challenge: | Existing methods to train relation extraction with distant supervision use noisy labels and implicitly assumes that all the KB facts are mentioned in the text. |
| Approach: | They propose to combine distant supervision data with additional directly-supervised data to train relation extraction models by using sigmoidal attention weights with max pooling. |
| Outcome: | The proposed method achieves state-of-the-art on the widely used FB-NYT dataset. |
Tweet Stance Detection Using an Attention based Neural Ensemble Model (N19-1)
Copied to clipboard
| Challenge: | Existing deep learning approaches to stance detection in twitter are inadequate to deal with the vanishing-gradient and overfitting problems. |
| Approach: | They propose a neural ensemble model that adopts strengths of two LSTM variants to learn better long-term dependencies. |
| Outcome: | The proposed model improves on the existing deep learning models on single and multi-target stance detection datasets. |
Convolutional Interaction Network for Natural Language Inference (D18-1)
Copied to clipboard
| Challenge: | Attention-based neural models have achieved great success in natural language inference (NLI). |
| Approach: | They propose a general model to capture the interaction between two sentences, which can be an alternative to the attention mechanism for NLI. |
| Outcome: | The proposed model can capture complex interactions on three large datasets. |
Recurrent Attention Networks for Long-text Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to encoding long documents using self-attention have been limited by quadratic computational complexities and limited application in long text processing. |
| Approach: | They propose a long-document encoding model that allows the recurrent operation of self-attention. |
| Outcome: | The proposed model extracts global semantics in token-level and document-level representations, making it inherently compatible with both sequential and sequential tasks. |
Automatic Distractor Generation for Multiple Choice Questions in Standard Tests (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to generate distractors for multiple choice questions are expensive and time-consuming. |
| Approach: | They propose a question and answer guided distractor generation framework to automate distractors generated by domain experts. |
| Outcome: | The proposed model outperforms existing models and achieves state-of-the-art on a large-scale dataset. |
Attending via both Fine-tuning and Compressing (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that attention mechanisms can improve models' interpretation, but they are not explicable. |
| Approach: | They propose a framework consisting of a learner and a compressor to purify attention scores . they propose to fine-tune and compress the attention mechanism to obtain a more faithful explanation . |
| Outcome: | The proposed framework improves performance and interpretability on eight benchmark datasets. |
t-HNE: A Text-guided Hierarchical Noise Eliminator for Multimodal Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis assume that all modalities contribute equally to model performance. |
| Approach: | They propose a text-guided Hierarchical Noise Eliminator model that extracts modality-consistent information from unimodal data and integrates it into multimodal representations for sentiment classification. |
| Outcome: | The proposed model reduces noise caused by modality inconsistency by maximizing mutual information between textual representations and visual and acoustic representations. |
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large pre-trained models have achieved outstanding results in sequence modeling . alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address these inefficiencies. |
| Approach: | They propose to reduce the size and computational overhead of large pre-trained models by removing selected components at different granularities. |
| Outcome: | The proposed models achieve a speedup of up to 1.4x during inference while maintaining accuracy. |
Efficient Large-Scale Neural Domain Classification with Personalized Attention (P18-1)
Copied to clipboard
| Challenge: | Using a scalable neural model, we show that personalization improves domain classification accuracy in a setting with thousands of overlapping domains. |
| Approach: | They propose a scalable neural model architecture with a shared encoder that incorporates personalization information and domain-specific classifiers that solves the problem efficiently. |
| Outcome: | The proposed architecture achieves two orders of magnitude faster than full model retraining. |
Skim-Attention: Learning to Focus via Document Layout (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to document understanding have high computational and memory costs. |
| Approach: | They propose a new attention mechanism that takes advantage of the structure of a document and its layout. |
| Outcome: | The proposed attention mechanism obtains lower perplexity than previous studies while being more computationally efficient. |
Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent research shows that attention heads are not confident in their decisions and can be pruned. |
| Approach: | They apply the lottery ticket hypothesis to prune heads in early training . they find that the pruned model is 1.5 times faster at inference . |
| Outcome: | The proposed method is 1.5 times faster at inference, but at the cost of longer training. |
Improvement in Sign Language Translation Using Text CTC Alignment (2025.coling-main)
Copied to clipboard
| Challenge: | Current sign language translation (SLT) approaches rely on gloss-based supervision with Connectionist Temporal Classification (CTC) limiting their ability to handle non-monotonic alignments between sign language video and spoken text. |
| Approach: | They propose a method that integrates CTC/Attention with the attention mechanism during decoding and integrates it with the sign language video and spoken text. |
| Outcome: | The proposed method outperforms the pure-attention baseline and achieves comparable results to state-of-the-art methods. |
Ruleformer: Context-aware Rule Mining over Knowledge Graph (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work on rule mining focuses on mining rules, but how to select appropriate rules for completion of different triplets has not been discussed. |
| Approach: | They propose to take context information into consideration when selecting suitable rules . they devise a transformer-based rule mining approach, Ruleformer . |
| Outcome: | The proposed model takes context information into consideration, which helps select suitable rules for inference tasks. |
Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction (P19-1)
Copied to clipboard
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, Junji Tomita
| Challenge: | Question answering (QA) using textual sources for purposes such as reading comprehension has attracted much attention. |
| Approach: | They propose a Query Focused Extractor model for evidence extraction and multi-task learning with the QA model. |
| Outcome: | The proposed model achieves state-of-the-art evidence extraction score on hotpotQA and FEVER, which is a recognizing textual entailment task on a large textual database. |
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | Recent results show pre-trained language models (LMs) can improve machine reading comprehension (MRC) Experimental results indicate that KT-NET offers significant and consistent improvements over BERT . |
| Approach: | They propose a method that leverages external knowledge bases to improve machine reading comprehension (MRC) KT-NET employs an attention mechanism to select desired knowledge from KBs and fuses selected knowledge with BERT to enable context- and knowledge-aware predictions. |
| Outcome: | The proposed model outperforms baseline models on ReCoRD and SQuAD1.1 benchmarks and ranks 1st on the ReCoDR and SQUAD1.1 leaderboards. |
ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on Multi-modal Named Entity Recognition (MNER) relies on image information to model interactions between image and text representations. |
| Approach: | They propose to align image features into the textual space to better utilize attention mechanisms . they use regional object tags, captions and optical characters as visual contexts . |
| Outcome: | The proposed model can achieve state-of-the-art accuracy on multi-modal Named Entity Recognition datasets even without image information. |
DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Temporal Language Grounding (TLG) is a task to determine temporal boundaries of video moments that correspond to a language query. |
| Approach: | They propose an energy-based model framework to explicitly learn moment-query distributions. |
| Outcome: | The proposed model outperforms the state-of-the-art models on four public temporal language grounding datasets. |
Cold-Start Aware User and Product Attention for Sentiment Classification (P18-1)
Copied to clipboard
| Challenge: | Existing models do not deal with cold-start problem typical in review websites. |
| Approach: | They propose a Hybrid Contextualized Sentiment Classifier that uses word encoder and Cold-Start Aware Attention to pool word vectors. |
| Outcome: | The proposed model performs significantly better on famous datasets despite having less complexity and can be trained much faster. |
Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking (2024.findings-acl)
Copied to clipboard
Zefeng Zhang, Jiawei Sheng, Zhang Chuang, Liangyunzhi Liangyunzhi, Wenyuan Zhang, Siqi Wang, Tingwen Liu
| Challenge: | Existing methods to link ambiguous mentions to entities in multimodal knowledge graphs rely on partial correlations. |
| Approach: | They propose a framework that leverages multi-element correlations to bridge modality gap and enable fine-grained semantic matching by exploiting correlation between multimodal features and entities. |
| Outcome: | The proposed framework outperforms state-of-the-art models and confirms the effectiveness of the proposed method. |
Encoding and Fusing Semantic Connection and Linguistic Evidence for Implicit Discourse Relation Recognition (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies use one attention mechanism to improve contextual semantic representation learning for implicit discourse relation recognition (IDRR). |
| Approach: | They propose a Multi-Attentive Neural Fusion model to fuse linguistic evidence and semantic connection for IDRR by using a Dual Attention Network and an Offset Matrix Network. |
| Outcome: | The proposed model achieves state-of-the-art on the PDTB 3.0 corpus. |
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents (2024.acl-long)
Copied to clipboard
| Challenge: | (large) language models struggle to process long sequences due to the quadratic scaling of the underlying attention mechanism. |
| Approach: | They propose a Masked Language Model operating on higher-level semantic representations in the form of text embeddings to solve this problem. |
| Outcome: | The proposed model outperforms larger embedding models on three types of tasks. |
Let’s do it “again”: A First Computational Approach to Detecting Adverbial Presupposition Triggers (P18-1)
Copied to clipboard
| Challenge: | a novel task of predicting adverbial presupposition triggers is useful for natural language generation . a focus is on a new attention mechanism for predicting presuposition trigger . |
| Approach: | They propose a new attention mechanism for predicting adverbial presupposition triggers . they propose to augment a baseline neural network without additional trainable parameters . |
| Outcome: | The proposed model outperforms baseline models in predicting adverbial presupposition triggers. |
Uncertainty Guided Global Memory Improves Multi-Hop Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Transformers are used to solve multi-hop question answering tasks that require reasoning over multiple parts of a long document. |
| Approach: | They propose a method that collects relevant information over the entire document and then combines it with local context to solve a multi-hop question answering task. |
| Outcome: | The proposed method improves on three MHQA datasets compared to the baseline model. |
TreeMAN: Tree-enhanced Multimodal Attention Network for ICD Coding (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to automatically assign ICD codes ignore crucial information contained in structured medical data, which is hard to be captured from the noisy clinical notes. |
| Approach: | They propose to use a Tree-enhanced multimodal attention network to fuse tabular features and textual features into multimodal representations by enhancing the text representations with tree-based features. |
| Outcome: | The proposed method outperforms state-of-the-art methods on two MIMIC datasets. |
Semi-supervised Entity Alignment via Joint Knowledge Embedding Model and Cross-graph Model (D19-1)
Copied to clipboard
| Challenge: | Entity alignment aims at integrating complementary knowledge graphs (KGs) from different sources or languages. |
| Approach: | They propose a semi-supervised entity alignment method by joint Knowledge Embedding model and Cross-Graph model to make better use of seed alignments to propagate over the entire graphs with KG-based constraints. |
| Outcome: | The proposed method can make better use of seed alignments to propagate over entire graphs with KG-based constraints. |
Semi-supervised New Event Type Induction and Description via Contrastive Loss-Enforced Batch Attention (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for event extraction use annotated event types but are expensive and time-consuming. |
| Approach: | They propose a semi-supervised approach to learning new event types using a masked contrastive loss. |
| Outcome: | The proposed method learns similarities between clusters by enforcing an attention mechanism over the data minibatch. |
RHO: Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge-grounded dialogue systems generate accurate and informative responses, but they are prone to hallucination problems. |
| Approach: | They propose a method to generate hallucinated responses using knowledge graphs . they propose local knowledge grounding to combine textual embeddings with corresponding KG embeddments . a global knowledge ground technique is also proposed to equip RHO with multi-hop reasoning abilities . |
| Outcome: | The proposed approach outperforms state-of-the-art methods on automatic and human evaluation by a large margin. |
Incorporating Word Attention into Character-Based Word Segmentation (N19-1)
Copied to clipboard
Shohei Higashiyama, Masao Utiyama, Eiichiro Sumita, Masao Ideuchi, Yoshiaki Oida, Yohei Sakamoto, Isaac Okada
| Challenge: | Word segmentation models are used to minimize the effort in feature engineering. |
| Approach: | They propose a character-based model that learns the importance of multiple candidate words for a corresponding character on the basis of an attention mechanism and makes use of it for segmentation decisions. |
| Outcome: | The proposed model outperforms the state-of-the-art models on Japanese and Chinese benchmark datasets. |
Interactive Key-Value Memory-augmented Attention for Image Paragraph Captioning (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to image paragraph captioning ignore the past alignment information, resulting in repetitive captioning and incomplete captioning. |
| Approach: | They propose an Interactive key-value Memory-augmented Attention model for image paragraph captioning to keep track of attention history along with update-chain of decoder state. |
| Outcome: | Extensive experiments on a benchmark dataset demonstrate the effectiveness of the proposed model. |
Distinguish Confusing Law Articles for Legal Judgment Prediction (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods to assist legal judgment are limited and can't solve confusing charges issue. |
| Approach: | They propose an end-to-end model to predict a legal judgment based on a textual description of the case and a graph neural network to learn subtle differences between confusing law articles. |
| Outcome: | The proposed model can learn subtle differences between confusing law articles and extract effective discriminative features from fact descriptions. |
Graph-based Fake News Detection using a Summarization Technique (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to detect fake news using external information and internal information are difficult to identify external information in all documents. |
| Approach: | They propose a graph-based fake news detection method that uses only the document internal information to represent the relationship between all sentences using a diagram and the reflection rate of contextual information among sentences is computed by using an attention mechanism. |
| Outcome: | The proposed method achieves high accuracy, 91.04%, that is 8.85%p better than the previous method. |
Ranking-Based Autoencoder for Extreme Multi-label Classification (N19-1)
Copied to clipboard
| Challenge: | Existing methods to solve label dependency and noisy labeling problems are limited . experimental results show the proposed method is competitive to state-of-the-art methods . |
| Approach: | They propose a deep learning XML method with word-vector-based self-attention followed by ranking-based AutoEncoder architecture to solve these problems. |
| Outcome: | The proposed method is competitive to state-of-the-art methods on benchmark datasets. |
H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences (2021.acl-long)
Copied to clipboard
| Challenge: | Existing hierarchical methods to compute attention are superior to sub-quadratic ones . a particular type of attention, called multi-head scaled dot-product attention, is one of the main components of the Transformer architecture . |
| Approach: | They propose a hierarchical method to compute attention in the Transformer architecture . they perform extensive experiments to show that it captures hierarchic structure in sequences . |
| Outcome: | The proposed method outperforms sub-quadratic models on the Long Range Arena benchmark by over +6 points on average. |
TAN-NTM: Topic Attention Networks for Neural Topic Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | Topic models have been widely used to learn text representations and gain insight into document corpora. |
| Approach: | They propose a framework which processes document as a sequence of tokens through a LSTM whose contextual outputs are attended in a topic-aware manner. |
| Outcome: | The proposed model improves on two downstream tasks: document classification and topic guided keyphrase generation. |
Contrastive Attention Mechanism for Abstractive Sentence Summarization (D19-1)
Copied to clipboard
| Challenge: | Existing attention mechanisms for abstractive sentence summarization are based on rule-based methods and large-scale training corpora. |
| Approach: | They propose a contrastive attention mechanism that extends the sequence-to-sequence framework for abstractive sentence summarization task. |
| Outcome: | The proposed mechanism improves the state-of-the-art on the abstractive sentence summarization task. |
Understanding Attention for Text Classification (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies have focused on whether local attention weights reflect the importance of input representations. |
| Approach: | They propose to analyze for each word token the following two quantities: its polarity score and its attention score, where the latter is a global assessment on the token’s significance. |
| Outcome: | The proposed model can be improved under conditions where the interplay between the two quantities can contribute towards model performance. |
Attention Mechanism with Energy-Friendly Operations (2022.findings-acl)
Copied to clipboard
| Challenge: | Empirical results show that attention mechanism can be improved from the energy consumption aspects. |
| Approach: | They propose to replace multiplications with either selective operations or additions to reduce energy consumption. |
| Outcome: | The proposed model achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure. |
Resource-Enhanced Neural Model for Event Argument Extraction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on event argument extraction (EE) is limited due to data scarcity and lack of a model encoder. |
| Approach: | They propose to capture the long-range dependency between an event trigger and a distant event argument using unlabeled data. |
| Outcome: | Experiments on the English ACE 2005 benchmark show that the proposed method achieves a new state-of-the-art. |
Sequence-to-Sequence Learning for Task-oriented Dialogue with Dialogue State Representation (C18-1)
Copied to clipboard
| Challenge: | Existing pipeline models for task-oriented dialogue system require explicit modeling of dialogue states and hand-crafted action spaces to query domain-specific knowledge base. |
| Approach: | They propose a framework that leverages the advantages of classic pipeline and sequence-to-sequence models. |
| Outcome: | The proposed framework outperforms baseline models on automatic and human evaluation on a Stanford Multi-turn Multi-domain task-oriented dialogue dataset. |
Integrating Tree Structures and Graph Structures with Neural Networks to Classify Discussion Discourse Acts (C18-1)
Copied to clipboard
| Challenge: | Existing models that analyze textual contents and discussion structures require understanding of textual content and discussion structure. |
| Approach: | They propose a model that integrates discussion structures with neural networks to classify discourse acts. |
| Outcome: | The proposed model improves accuracy and FB1 score by 1.5% compared to the previous best model. |
Attention weights accurately predict language representations in the brain (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In Transformer-based language models, the attention mechanism converts token embeddings into contextual embeddables that incorporate information from neighboring words. |
| Approach: | They analyze fMRI recordings of English language learners and extract attention weights from them to determine how well they can predict brain responses. |
| Outcome: | The resulting hidden state embeddings are more accurate than lexical embeddngs or RNN-based models. |
Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown strong potential for tabular data generation by modeling textualized feature-value pairs. |
| Approach: | They propose a method that explicitly integrates sparse dependency graphs into LLMs’ attention mechanism. |
| Outcome: | The proposed method outperforms existing LLM-based approaches by up to 12% on complex datasets while achieving competitive results with state-of-the-art approaches in synthetic data quality. |
Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Neural Machine Translation models treat decoding at each time step equally with the same matrix . conventional methods treat decoder outputs at all time steps with the identical weight matrix causing inaccuracy . |
| Approach: | They propose a model with a mechanism to control the softness of attention by means of an attention temperature. |
| Outcome: | The proposed model outperforms baseline models on Chinese-English and English-Vietnamese translations. |
SPE Attention: Making Attention Equivariant to Semantic-Preserving Permutation for Code Processing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train code processing models to capture symmetry of code . semantic-preserving permutations are not found in natural languages . |
| Approach: | They propose a mechanism that captures a unique symmetry of code, called the SPE attention . they propose symmetry graphs that are then combined to create a symmetry mask . |
| Outcome: | The proposed model can be used to analyze code summarization and error detection tasks. |
Training Deeper Neural Machine Translation Models with Transparent Attention (D18-1)
Copied to clipboard
| Challenge: | Existing NMT models are shallow in comparison to convolutional models used for both text and vision tasks. |
| Approach: | They propose to modify the attention mechanism to ease the optimization of deeper models by a simple modification to the seq2seq with attention paradigm. |
| Outcome: | The proposed model achieves consistent gains of 0.7-1.1 BLEU on the benchmark WMT’14 English-German and WMT'15 Czech-English tasks. |
Dependency-driven Relation Extraction with Attentive Graph Convolutional Networks (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies suffer from noise in dependency trees, which can cause confusions in relation extraction. |
| Approach: | They propose a dependency-driven approach for relation extraction with attentive graph convolutional networks . they apply an attention mechanism upon graph convolutional networks to different word dependencies . |
| Outcome: | The proposed approach outperforms previous studies on two English datasets and achieves state-of-the-art performance. |
Irony Detection in Persian Language: A Transfer Learning Approach Using Emoji Prediction (2020.lrec-1)
Copied to clipboard
Preni Golazizian, Behnam Sabeti, Seyed Arad Ashrafi Asli, Zahra Majdabadi, Omid Momenzadeh, Reza Fahmi
| Challenge: | Existing methods for emotion extraction and sentiment analysis produce invalid results due to the use of irony. |
| Approach: | They propose to use emoji prediction to fine tune a model using hand labeled tweets with irony tags. |
| Outcome: | The proposed method outperforms the state-of-the-art method on Persian dataset with an accuracy of 83.1% and offers strong baseline for further research in Persian language. |
Complex Hyperbolic Knowledge Graph Embeddings with Fast Fourier Transform (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that the choice of space for knowledge graph (KG) embeddings has significant effects on the performance of KG completion tasks. |
| Approach: | They propose to use the Fourier transform to convert between real and complex hyperbolic space to capture hierarchical patterns. |
| Outcome: | The proposed models outperform the baseline models for knowledge graph (KG) embeddings. |
SimulSpeech: End-to-End Simultaneous Speech to Text Translation (2020.acl-main)
Copied to clipboard
| Challenge: | SimulSpeech is an end-to-end simultaneous speech to text translation system . conventional approaches to simultaneous speech translation divide the translation process into two stages . |
| Approach: | They develop an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently. |
| Outcome: | The proposed system achieves reasonable BLEU scores and lower delay compared to full-sentence translation model. |
LittleBird: Efficient Faster & Longer Transformer for Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for BERT have a limitation dealing with long inputs due to its attention mechanism. |
| Approach: | They propose a model based on BigBird with improved speed and memory footprint . they propose 'pack and unpack attention' to replace global attention . |
| Outcome: | The proposed model can work on long inputs even after being pre-trained on short inputs. |
Parallel Context Windows for Large Language Models (2023.acl-long)
Copied to clipboard
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham
| Challenge: | Existing efforts to address context window limitation for off-the-shelf LLMs involve training specialized architectures. |
| Approach: | They propose a method that carves a long context into chunks and restricts attention to apply only within each window. |
| Outcome: | The proposed method shows significant improvements on in-context learning tasks with diverse input and output spaces. |
How Knowledge Graph and Attention Help? A Qualitative Analysis into Bag-level Relation Extraction (2021.acl-long)
Copied to clipboard
| Challenge: | Knowledge Graph (KG) and attention mechanism have been demonstrated effective in introducing and selecting useful information for weakly supervised methods. |
| Approach: | They propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE) they propose to incorporate entity prior to KG-enhanced attention to improve RE performance . |
| Outcome: | The proposed model achieves significant improvements on two real-world datasets compared with three state-of-the-art baselines. |
ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation (P19-1)
Copied to clipboard
| Challenge: | Existing hierarchical recurrent encoder-decoder models treat all contexts indiscriminately, which may hurt the following response generation process. |
| Approach: | They propose a hierarchical recurrent encoder-decoder model that treats all contexts indiscriminately and uses a word level LSTM encoder to obtain the initial representation of each context. |
| Outcome: | The proposed model outperforms baseline models on Chinese customer services and English Ubuntu dialogue datasets in terms of both metric-based and human evaluations. |
Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work shows that deeper character-based neural machine translation models outperform subword-based models. |
| Approach: | They propose to investigate the ability of character-based models to learn word senses and morphological inflections and the attention mechanism in Finnish into English translation. |
| Outcome: | The character-based models outperform subword-based model in Finnish to English translation. |
Knowledge Enhanced Masked Language Model for Stance Detection (2021.naacl-main)
Copied to clipboard
| Challenge: | Detecting stance on Twitter is difficult because of the short length of each tweet . Twitter content is dynamic, constantly coining new terminology and hashtags . |
| Approach: | They propose a BERT-based fine-tuning method that enhances stance detection models . they use weighted log-odds-ratio to identify words with high stance distinguishability . |
| Outcome: | The proposed method outperforms the state-of-the-art for stance detection on Twitter data about the 2020 US presidential election. |
LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing retrieval-based methods compromise semantic integrity through fixed-size chunking and suffer from inefficient linear scanning. |
| Approach: | They propose a method that preserves local semantic coherence through boundary-aware chunking and constructs a recursive hierarchical index rooted in the triangle inequality. |
| Outcome: | The proposed method achieves 3.6 end-to-end inference speedup with negligible degradation in model performance. |
Thread Popularity Prediction and Tracking with a Permutation-invariant Model (D18-1)
Copied to clipboard
| Challenge: | a task of thread popularity prediction and tracking aims to recommend a few popular comments to subscribed users when a batch of new comments arrive in a discussion thread. |
| Approach: | They propose a deep neural network architecture to model the expected cumulative reward of a recommendation (action) they employ a greedy procedure to approximate the action that maximizes the predicted Q-value . |
| Outcome: | The proposed approach outperforms the state-of-the-art on five real-world datasets. |
Query-Key Normalization for Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Low-resource language translation is a challenging but socially valuable NLP task. |
| Approach: | They propose a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. |
| Outcome: | The proposed technique improves 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT’15. |
Too Big to Fail: Larger Language Models are Disproportionately Resilient to Induction of Dementia-Related Linguistic Anomalies (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that the attention mechanism in transformer-based NLMs may present an analogue to the notions of cognitive and brain reserve. |
| Approach: | They propose a bidirectional ablation method that masks attention heads to display degradation of similar magnitude to masking in smaller models. |
| Outcome: | The proposed method exhibits properties attributed to the concepts of cognitive and brain reserve in human brain studies. |
Uni-Encoder: A Fast and Accurate Response Selection Paradigm for Generation-Based Dialogue Systems (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to decode a given context-candidate pair are expensive and time-consuming. |
| Approach: | They propose a new paradigm that keeps full attention over each pair while only encoding the context once. |
| Outcome: | The proposed paradigm achieves new state-of-the-art results on four benchmark datasets with high computational efficiency. |
Improving Distantly-Supervised Relation Extraction with Joint Label Embedding (D19-1)
Copied to clipboard
| Challenge: | Existing methods for relation extraction treat labels as independent and meaningless one-hot vectors, which cause a loss of potential label information for selecting valid instances. |
| Approach: | They propose a multi-layer attention-based model to improve relation extraction with joint label embedding by gating integration and using the embeddable entities as an atten- tion. |
| Outcome: | The proposed model significantly outperforms state-of-the-art methods in relation extraction with joint label embedding. |
Query-Aware Graph Attention for Precise Subgraph Retrieval in Knowledge-Augmented Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing Knowledge Graph (KG)-based Retrieval-Augmented Generation (RAG) systems insufficiently model the interaction between query semantics and relation types, resulting in imprecise subgraph retrieval and unstable reasoning. |
| Approach: | They propose a retrieval framework that integrates query semantics and relation embeddings directly into the attention mechanism. |
| Outcome: | Experiments on WebQSP and CWQ establish new state-of-the-art results in both Triple Recall and Answer Recall. |
Large Sequence Representation Learning via Multi-Stage Latent Transformers (2022.coling-1)
Copied to clipboard
| Challenge: | a novel algorithm for named-entity recognition (NER) uses language and spatial features to predict entity tags for structured text . a dataset of 11,926 images depicting food product labels is used to perform NER tasks . |
| Approach: | They propose a multi-stage transformer architecture for named-entity recognition . they propose RADAR, an LSTM classifier operating at character level, to refine NER predictions . |
| Outcome: | The proposed method outperforms two competing models on a food label dataset. |
Combinatory Grammar Tells Underlying Relevance among Entities (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches focus on dependencies among words while paying limited attention to other types of syntactic structure. |
| Approach: | They propose an alternative approach that takes advantage of combinatory categorial grammar to detect the relation between entities. |
| Outcome: | The proposed model performs state-of-the-art on two widely used English benchmark datasets. |
Detecting Sexual Content at the Sentence Level in First Millennium Latin Texts (2024.lrec-main)
Copied to clipboard
| Challenge: | a traditional approach to corpus building involves constructing a corpus centered around specific themes, such as colors. |
| Approach: | They propose to use deep learning methods to accelerate corpus building in humanities . they propose to integrate metadata embeddings into the model to improve accuracy . |
| Outcome: | The proposed method outperforms token-based searches in the humanities and linguistics field. |
Sentence-aware Adversarial Meta-Learning for Few-Shot Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies fail to consider the importance of the semantic interaction between sentence features and neglect to enhance the generalization ability of the model to new tasks. |
| Approach: | They propose to integrate an adversarial network architecture into the meta-learning system and leverage cost-effective modules to build a few-shot classification framework called SaAML. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on four benchmark datasets. |
A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on how large language models process and store information related to arithmetic tasks have shown their behavior inconsistent and context-dependent. |
| Approach: | They propose to mechanize the processing of arithmetic queries by a causal mediation framework. |
| Outcome: | The proposed model improves the performance of arithmetic queries with a set of MLP modules. |
Generalized Supervised Attention for Text Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing supervised attention methods that use human knowledge to learn better alignments are costly or infeasible. |
| Approach: | They propose a generalized supervised attention method based on quasi alignments that are easier to obtain than ideal alignments. |
| Outcome: | The proposed framework improves generation performance and is robust against errors in attention supervision. |
Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel (D19-1)
Copied to clipboard
| Challenge: | Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. |
| Approach: | They propose a new formulation of attention via the lens of the kernel which allows us to understand individual components of Transformer's attention. |
| Outcome: | The proposed model outperforms existing models on language understanding and sequence prediction tasks and is more efficient than existing models. |
AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for augmented large language models suffer from irrelevant retrieved content . existing methods struggle to adapt compression rates for different context, maintain low latency . |
| Approach: | We propose an adaptive, efficient and context-aware compression framework to reduce retrieved content . AttnComp uses a top-p compression algorithm to retain the minimal set of documents whose attention weights exceed a threshold. |
| Outcome: | Experiments show that AttnComp outperforms existing compression methods and uncompressed baselines in achieving higher accuracy with substantial compression rates and lower latency. |
Rewarding Smatch: Transition-Based AMR Parsing with Reinforcement Learning (P19-1)
Copied to clipboard
| Challenge: | Abstract meaning representations (AMRs) are labeled directed acyclic graphs that represent a non intersentential abstraction of natural language with broad-coverage semantic representations. |
| Approach: | They build upon a transition-based AMR parser that uses Stack-LSTMs and augment training with policy learning. |
| Outcome: | The proposed parser performs comparable to the best published parsers. |
A Representation Level Analysis of NMT Model Robustness to Grammatical Errors (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work on robustness failures or improving robustness has focused on documenting failures . however, there has been limited analysis of model representations in response to noise. |
| Approach: | They perform Grammatical Error Detection probing and representational similarity analysis to examine model representations of ungrammatical inputs and how they evolve through model layers. |
| Outcome: | The proposed model detects and corrects the grammatical error by moving its representation toward the correct form. |
Multilingual Detection of Personal Employment Status on Twitter (2022.acl-long)
Copied to clipboard
| Challenge: | Detecting disclosures of individuals’ employment status on social media is a challenging task due to their rarity in a sea of social media content and the variety of linguistic forms used to describe them. |
| Approach: | They propose to use BERT-based classification models to identify five types of disclosures about individuals’ employment status in three languages. |
| Outcome: | The proposed methods achieve significant gains in precision, recall, and diversity of results in real-world settings of extreme class imbalance. |
DocLLM: A Layout-Aware Generative Language Model for Multimodal Document Understanding (2024.acl-long)
Copied to clipboard
Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, Xiaomo Liu
| Challenge: | Documents with rich layouts are a significant portion of enterprise corpora and document AI is still a challenge. |
| Approach: | They propose a lightweight extension to traditional large language models for reasoning over visual documents that takes into account both textual semantics and spatial layout. |
| Outcome: | The proposed model outperforms existing large language models on 14 out of 16 datasets and generalizes well to 4 out of 5 previously unseen datasets. |
Improving Non-Autoregressive Neural Machine Translation via Modeling Localness (2022.coling-1)
Copied to clipboard
| Challenge: | Existing non-autoregressive neural machine translation models suffer from poor localization quality due to sequential dependencies within the target sentence. |
| Approach: | They propose to introduce local information into NAT models by explicitly introducing local information about surrounding words into the encoder and decoder sides to achieve localness-aware representations. |
| Outcome: | The proposed method can achieve significant improvements over strong NAT baselines. |
Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study explores efficient ultra-long context modeling. |
| Approach: | They propose to use Hierarchical Sparse Attention to achieve efficient ultra-long context modeling. |
| Outcome: | The proposed model performs comparable to full-attention baselines on in-domain and out-of-domain tasks. |
Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks (D19-1)
Copied to clipboard
| Challenge: | Existing aspects-based sentiment classification models lack a mechanism to account for relevant syntactical constraints and word dependencies. |
| Approach: | They propose to build a Graph Convolutional Network over the dependency tree of a sentence to exploit syntactical information and word dependencies. |
| Outcome: | The proposed model is comparable to state-of-the-art models on three benchmarking collections. |
CAN: Constrained Attention Networks for Multi-Aspect Sentiment Analysis (D19-1)
Copied to clipboard
| Challenge: | Existing methods for aspect-specific sentiment classification are noisy and downgraded performance. |
| Approach: | They propose a constrained attention network to regularize attention for multi-aspect sentiment analysis by orthogonal regularization on multiple aspects and sparse regularization for each single aspect. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on two public datasets and extends to multi-task settings. |
Different Absorption from the Same Sharing: Sifted Multi-task Learning for Fake News Detection (D19-1)
Copied to clipboard
| Challenge: | Existing methods for detecting fake news use shared features as complementarity features without selection. |
| Approach: | They propose a sifted multi-task learning method with a selected sharing layer for fake news detection. |
| Outcome: | The proposed method boosts the F1-score by more than 0.87%, 1.31% on two public and widely used competition datasets. |
Hard Non-Monotonic Attention for Character-Level Transduction (D18-1)
Copied to clipboard
| Challenge: | Character-level string-to-string transductions are an important component of NLP tasks . hard non-monotonic attention models have been used for sequence modeling tasks involving characters . |
| Approach: | They propose an exact algorithm for marginalizing over the exponential number of non-monotonic alignments between two strings. |
| Outcome: | The proposed algorithm outperforms soft attention and improves performance over stochastic approximation. |
Symmetric Dot-Product Attention for Efficient Training of BERT Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Transformer-based models are stretched to enormous sizes, requiring increasingly larger training datasets and unsustainable amount of compute resources. |
| Approach: | They propose an alternative compatibility function for the Transformer-based attention mechanism that exploits an overlap in the learned representation of the traditional scaled dot-product attention mechanism. |
| Outcome: | The proposed model achieves 79.36 on the GLUE benchmark against 78.74 for the traditional implementation and reduces the number of trainable parameters by 6%. |
If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation? (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work in computational psycholinguistics has revealed intriguing parallels between attention mechanisms and human memory retrieval, focusing primarily on vanilla Transformers that operate on token-level representations. |
| Approach: | They propose that the attention mechanism of Transformer Grammar (TG) can serve as a cognitive model of human memory retrieval using Normalized Attention Entropy (NAE) they propose that TG's attention can implement a human memory-retrieval theory known as cue-based retrieval . |
| Outcome: | The attention mechanism of Transformer Grammar (TG) achieves superior predictive power for self-paced reading times compared to vanilla Transformer’s, with further analyses revealing independent contributions from both models. |
Mix-Initiative Response Generation with Dynamic Prefix Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing dialogue systems focus on training a holistic response generation model without any distinction between different initiatives. |
| Approach: | They propose a general mix-Initiative Dynamic Prefix Tuning framework to decouple different initiatives from the generation model. |
| Outcome: | The proposed framework outperforms baselines on two public dialogue datasets on human evaluations and automatic metrics. |
Multi-Layer Pseudo-Siamese Biaffine Model for Dependency Parsing (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work only uses biaffine method at the end of the dependency parser as a scorer, and its application in multi-layer form is ignored. |
| Approach: | They propose a multi-layer pseudo-Siamese biaffine model for neural dependency parsing that uses biaffin method as a scorer and a biaffin module to construct arc weight matrix. |
| Outcome: | The proposed model achieves state-of-the-art on PTB, CTB, and UD datasets with low efficiency loss. |
Joint Biomedical Entity and Relation Extraction with Knowledge-Enhanced Collective Inference (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for information extraction from biomedical texts do not utilize external knowledge . despite the exponential growth of biomedically published articles, many existing methods fall behind . |
| Approach: | They propose a framework that utilizes external knowledge for entity and relation extraction . KECI uses an initial span graph to construct a knowledge graph containing relevant background knowledge . |
| Outcome: | The proposed framework achieves state-of-the-art results in two biomedical datasets . it achieves 4.59% and 4.91% improvement in F1 scores over the state- of-the art methods . |
Heterogeneous Graph Attention Networks for Semi-supervised Short Text Classification (D19-1)
Copied to clipboard
| Challenge: | Existing studies on short text classification focus on long texts and achieve unsatisfactory performance due to the sparsity and limited labeled data. |
| Approach: | They propose a heterogeneous graph neural network based method for semi-supervised short text classification that leverages the full advantage of few labeled data and large unlabeled data through information propagation along the graph. |
| Outcome: | The proposed method outperforms state-of-the-art methods across six benchmark datasets significantly. |
Reviews Meet Graphs: Enhancing User and Item Representations for Recommendation with Hierarchical Attentive Graph Neural Network (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn user and item representations from review texts do not take into account the user-user and item-item relatedness of the user. |
| Approach: | They propose to use review content and user-item graphs to integrate them as different views. |
| Outcome: | The proposed approach can learn user and item representations from review content and user-item graphs. |
Dual Capsule Attention Mask Network with Mutual Learning for Visual Question Answering (2022.coling-1)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) models extract features from images and questions independently, but these methods fail to capture fine-grained key features and include much unnecessary information. |
| Approach: | They propose a dual capsule attention mask network with mutual learning for visual question answering (VQA) it contains two branches processing coarse-grained features and fine-grain features, respectively. |
| Outcome: | The proposed model outperforms baselines in terms of performance and interpretability and achieves new SOTA performance on the VQA-v2 dataset. |
Hierarchical Transformers for Multi-Document Summarization (P19-1)
Copied to clipboard
| Challenge: | Existing models for multidocument summarization have been developed that can process multiple documents in a hierarchical manner. |
| Approach: | They propose a neural summarization model which can process multiple input documents and distill Transformer architecture with the ability to encode documents in a hierarchical manner. |
| Outcome: | The proposed model improves on the WikiSum dataset and can process multiple documents in a hierarchical manner. |
CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning paradigms focus on aligning LLMs with task-specific objectives. |
| Approach: | They propose a pipeline that leverages human priors to automatically generate token-level causal signals and introduce the Re-Attention mechanism to guide training. |
| Outcome: | The proposed pipeline achieves an average improvement of 5.76% on the STG dataset and 1.56% on downstream tasks. |
CoMix: Guide Transformers to Code-Mix using POS structure and Phonetics (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing multilingual transformer models lack the ability to intermix words of one language into the structure of another. |
| Approach: | They propose a pretraining approach to improve representation of code-mixed data in transformer models by incorporating phonetic signals, a modified attention mechanism and weak supervision guided generation by parts-of-speech constraints. |
| Outcome: | The proposed model improves performance across four code-mixed tasks and generalizes on out-of-domain translation. |
Online Conversation Disentanglement with Pointer Networks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for disentangling textual conversations rely on dataset specific features that hinder generalization and adaptability. |
| Approach: | They propose an end-to-end online framework for conversation disentanglement that embeds the whole utterance that comprises timestamp, speaker, and message text. |
| Outcome: | The proposed method performs state-of-the-art on the Ubuntu IRC dataset and on other social and organizational platforms. |
Scaling up Open Tagging from Tens to Thousands: Comprehension Empowered Attribute Value Extraction from Product Title (P19-1)
Copied to clipboard
| Challenge: | Existing models treat each attribute as an entity type and build one set of NER tags for each of them, leading to scalability issues. |
| Approach: | They propose to regard attribute as a query and adopt only one global set of BIO tags for any attributes to reduce the burden of attribute tag or model explosion. |
| Outcome: | The proposed model outperforms state-of-the-art models and generates promising results for 8,906 attributes. |
ABC: Attention with Bounded-memory Control (2022.acl-long)
Copied to clipboard
Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, Noah A. Smith
| Challenge: | Existing approaches to attention with bounded-memory control (ABC) have a quadratic complexity in sequence lengths, making it prohibitive for long sequences. |
| Approach: | They propose a new abstraction that bounds memory size to improve efficiency . they propose bounded-memory control, which connects several efficient attention variants . |
| Outcome: | The proposed approach outperforms existing approaches on language modeling, machine translation, and masked language model finetuning. |
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation (2025.acl-long)
Copied to clipboard
Chuanyang Zheng, Yihang Gao, Han Shi, Jing Xiong, Jiankai Sun, Jingyao Li, Minbin Huang, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, Yu Li
| Challenge: | Extensive experiments demonstrate that treating attention as a feature map and applying convolution as . a processing method significantly enhances Transformer performance. |
| Approach: | They propose to use the convolution operator to mimic the processing methods in computer vision to treat attention as a feature map and apply it to neighboring attention scores across different heads. |
| Outcome: | The proposed model can be adapted to various attention-related models and achieves high performance. |
Interpreting Positional Information in Perspective of Word Order (2023.acl-long)
Copied to clipboard
| Challenge: | Attention mechanism is a powerful and effective method utilized in natural language processing, but it is insensitive to positional information. |
| Approach: | They propose a weight concatenation operation to evaluate its efficacy in machine translation tasks. |
| Outcome: | The proposed operation can encode positional information and confirms our hypothesis. |
Paraphrase Generation by Learning How to Edit from Samples (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show the superiority of our retrieval-based paraphrase generation model in terms of both automatic metrics and human evaluation of relevance, grammaticality, and diversity of generated paraphrases. |
| Approach: | They propose a retrieval-based method for paraphrase generation which uses a novel editor module to extract edits from paraphrase pairs. |
| Outcome: | The proposed model outperforms existing models in automatic metrics and human evaluation of relevance, grammaticality, and diversity of generated paraphrases. |
Paraphrase Generation Evaluation Powered by an LLM: A Semantic Metric, Not a Lexical One (2025.coling-main)
Copied to clipboard
| Challenge: | Existing measures for automatic paraphrase generation are based on lexical distances or semantic embedding alignments. |
| Approach: | They propose a measure based on a log likelihood ratio from an LLM to assess the quality of a potential paraphrase. |
| Outcome: | The proposed measure is better for sorting pairs of sentences by semantic proximity and provides an interpretable classification threshold between paraphrases and non-paraphrases. |
Fine-Grained Transfer Learning for Harmful Content Detection through Label-Specific Soft Prompt Tuning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing detection models are less effective and generalizable due to static data. |
| Approach: | They propose a method that leverages class-specific knowledge to enhance harmful content detection. |
| Outcome: | The proposed method improves harmful content detection across English and German datasets. |
Capsule Network with Interactive Attention for Aspect-Level Sentiment Classification (D19-1)
Copied to clipboard
| Challenge: | Existing methods for aspect-level sentiment classification are limited for dealing with overlapped features. |
| Approach: | They propose to use capsule network to construct vector-based feature representation and cluster features by an EM routing algorithm to model semantic relationship between aspect terms and context. |
| Outcome: | The proposed model achieves state-of-the-art on three datasets. |
Rethinking Attribute Representation and Injection for Sentiment Classification (D19-1)
Copied to clipboard
| Challenge: | Existing models that use text attributes to improve sentiment classification use text as a categorical feature. |
| Approach: | They propose to represent attributes as chunk-wise importance weight matrices and consider four locations to inject attributes. |
| Outcome: | The proposed method outperforms the state-of-the-art and outperformed previous models. |
EARA: Improving Biomedical Semantic Textual Similarity with Entity-Aligned Attention and Retrieval Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to measure semantic similarity between biomedical texts are inefficient due to too many biomedically-related entities. |
| Approach: | They propose an entity-aligned, attention-based and retrieval-augmented PLM that aligns the same type of fine-grained entity information in each sentence pair with an entity alignment matrix with an auxiliary loss. |
| Outcome: | The proposed model can achieve state-of-the-art on both in-domain and out-of domain datasets. |
Using Human Attention to Extract Keyphrase from Microblog Post (P19-1)
Copied to clipboard
| Challenge: | Existing studies on keyphrase extraction neglect human reading behavior during keyphrase annotating. |
| Approach: | They propose to integrate human attention into keyphrase extraction models by an attention mechanism and combine it with neural network models. |
| Outcome: | The proposed models improve on two Twitter datasets. |
Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification (2020.acl-main)
Copied to clipboard
| Challenge: | Aspect-based sentiment classification is a popular task aimed at identifying the corresponding emotion of a given aspect. |
| Approach: | They propose a dependency graph enhanced dual-transformer network to support mutual reinforcement between the flat representation learning and graph-based representation learning. |
| Outcome: | The proposed model outperforms state-of-the-art methods on five datasets with a large margin. |
Learning from Adjective-Noun Pairs: A Knowledge-enhanced Framework for Target-Oriented Multimodal Sentiment Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to determine sentiment polarity of opinion target are inconsistent and lack visual attention. |
| Approach: | They propose a framework which can exploit adjective-noun pairs extracted from images to improve visual attention and sentiment prediction capability of the TMSC task. |
| Outcome: | The proposed framework outperforms state-of-the-art on two public datasets. |
Improving NMT Quality Using Terminology Injection (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study has explored the use of vetted terminology in neural machine translation . a number of organizations use domain- or organization-specific words and phrases . |
| Approach: | They propose a method for injecting terminology and for evaluating terminology injection. |
| Outcome: | The proposed method is based on the long-term memory (LSTM) attention mechanism prevalent in state-of-the-art systems . it also introduces a new translation metric more sensitive to approved terminological content in MT output. |
Embedding Dynamic Attributed Networks by Modeling the Evolution Processes (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to embed nodes into low-dimensional vectors focus on static networks, but in practice, many networks are evolving over time and hence are dynamic, e.g., social networks. |
| Approach: | They propose to extract high-order neighborhood information at each given timestamp and then use an embedding prediction framework to capture the temporal correlations. |
| Outcome: | Extensive experiments on four real-world datasets show that the proposed method outperforms baseline methods for dynamic link prediction and node classification tasks. |
Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation (P19-1)
Copied to clipboard
| Challenge: | Disentangling the content and style in the latent space is prevalent in text style transfer . recurrent neural networks (RNN) based encoder and decoder cannot deal with the long-term dependency . |
| Approach: | They propose a style transformer which disentangles style information in latent space . they propose encoding and decoding methods that disentangle style information . |
| Outcome: | The proposed method can achieve better style transfer and better content preservation. |
Layer-Condensed KV Cache for Efficient Inference of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using a key-value cache, memory consumption is a bottleneck for high-throughput language models. |
| Approach: | They propose a method that only computes and caches the KVs of a small number of layers, thus saving memory consumption and improving inference throughput. |
| Outcome: | The proposed method achieves higher throughput and competitive performance than standard transformers and is orthogonal to existing transformer memory-saving techniques. |
Answering Conversational Questions on Structured Data without Logical Forms (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to answering sequential questions based on structured objects do not use a logical form as an intermediate representation. |
| Approach: | They propose a novel approach to answering sequential questions based on structured objects without using a logical form as an intermediate representation. |
| Outcome: | The proposed approach is competitively tested on the Sequential Question Answering (SQA) task. |
Attention Word Embedding (2020.coling-main)
Copied to clipboard
| Challenge: | Word embedding models learn semantically rich vector representations of words . popular word embedders include word2vec, GloVe, and fastText . |
| Approach: | They propose an AWE-S model which integrates the attention mechanism into the CBOW model and incorporates subword information. |
| Outcome: | The proposed model outperforms the state-of-the-art model on word similarity datasets and when used for initialization of NLP models. |
Revealing and Mitigating the Local Pattern Shortcuts of Mamba (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that Mamba excels in tasks that involve localized key information but faces challenges with tasks that require handling distributed key information. |
| Approach: | They propose to introduce a global gate module into Mamba to address this problem by adding 4M extra parameters to the model. |
| Outcome: | The proposed model outperforms attention-based models on synthetic and synthetic tasks with only 4M extra parameters. |
Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that multilingual language models (MultiLMs) are capable of logically reasoning over natural language statements, reasoning with their implicit knowledge, and performing multi-step reasoning when the model size is large enough. |
| Approach: | They propose a mechanism that encourages cross-lingual attention in code-switched sequences and improves reasoning performance by up to 14%. |
| Outcome: | The proposed approach improves reasoning performance by 14% and 4% on the RuleTaker and LeapOfThought datasets. |
An Attentive Fine-Grained Entity Typing Model with Latent Type Representation (D19-1)
Copied to clipboard
| Challenge: | Existing fine-grained entity typing models are criticized for label independence assumption . |
| Approach: | They propose a fine-grained entity typing model with a new attention mechanism and a hybrid type classifier to exploit type inter-dependency with latent type representation. |
| Outcome: | The proposed model significantly advances the state-of-the-art on fine-grained entity typing. |
Lattice Transformer for Speech Translation (P19-1)
Copied to clipboard
| Challenge: | Recent advances in sequence modeling have highlighted the strengths of the transformer architecture. |
| Approach: | They propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) they propose 'controllable' lattica attention mechanism to consume latent representations. |
| Outcome: | The proposed model outperforms baseline and lattice LSTM on the Chinese-English translation task. |
Multi-Task Stance Detection with Sentiment and Stance Lexicons (D19-1)
Copied to clipboard
| Challenge: | Recent studies show improvements in stance detection by using attention mechanism or sentiment information. |
| Approach: | They propose a multi-task framework that incorporates attention mechanism and takes sentiment classification as an auxiliary task. |
| Outcome: | The proposed model outperforms state-of-the-art deep learning methods on the SemEval-2016 dataset. |
From Graph to Word Bag: Introducing Domain Knowledge to Confusing Charge Prediction (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing charge prediction methods have shown impressive performance, but they face significant challenges when dealing with confusing charges, such as Snatch and Robbery. |
| Approach: | They propose a novel approach which introduces domain knowledge regarding constituent elements to guide the model in making judgments on confusing charges, much like a judge’s reasoning process. |
| Outcome: | The proposed approach maintains exceptional performance in imbalanced label distributions. |
GCNet: Global-and-Context Collaborative Learning for Aspect-Based Sentiment Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for analyzing aspect terms are focused on extracting semantic information inherent within the sentence. |
| Approach: | They propose a GCNet that explicitly leverages global semantic information to guide context encoding. |
| Outcome: | The proposed model outperforms state-of-the-art methods on three public datasets. |
Causality-Guided Multi-Memory Interaction Network for Multivariate Stock Price Movement Prediction (2023.acl-long)
Copied to clipboard
| Challenge: | Existing models for stock price movement prediction use auxiliary data, but we assume other stocks should be utilized as auxiliary information to enhance performance. |
| Approach: | They propose a Causality-guided multi-memory interaction network for stock movement prediction which transforms basic attention into Causal Attention by calculating transfer entropy between multivariate stocks. |
| Outcome: | The proposed model outperforms existing models on three real-world datasets from the U.S. and Chinese markets. |
Robertha: Eigenspectrum Regularized Attention for Robust Natural Language Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing robustness methods sacrifice clean performance or fail to generalize to higher corruption levels. |
| Approach: | They propose a mechanism that uses semantic patterns to pull corrupted embeddings toward correct representations by Eigenspectrum Regularization. |
| Outcome: | The proposed mechanism outperforms robustness methods on 13 GLUE and SuperGLUE tasks while maintaining competitive clean performance. |
How Do Hyenas Deal with Human Speech? Speech Recognition and Translation with ConfHyena (2024.lrec-main)
Copied to clipboard
| Challenge: | Currently, attention-based models face computational hurdles in processing long sequences due to its quadratic complexity. |
| Approach: | They propose a conformer whose encoder self-attentions are replaced with Hyena for speech processing . they propose 'confhyena' model that reduces training time by 27% at minimal cost . |
| Outcome: | The proposed model reduces training time by 27% at the cost of minimal quality degradation. |
Aspect-to-Scope Oriented Multi-view Contrastive Learning for Aspect-based Sentiment Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Aspect-based sentiment analysis (ABSA) focus on mining syntactic or semantic information, which suffers from noisy interference when multiple aspects exist in a sentence. |
| Approach: | They propose a scope-assisted multi-view graph contrastive learning framework that captures correlation and difference between aspect and syntactic/semantic information. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on five benchmark datasets and verifies its effectiveness and robustness. |
Chain and Causal Attention for Efficient Entity Tracking (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to handle entity tracking require at least log2 (n+1) layers to handle n state changes. |
| Approach: | They propose an efficient enhancement to the standard attention mechanism to handle long-term dependencies with a single layer. |
| Outcome: | The proposed model can handle entity tracking with n state changes with a single layer. |
Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems (2021.emnlp-main)
Copied to clipboard
| Challenge: | Training and inference using large transformer models can be computationally expensive because the self-attention's time and memory grow quadratically with sequence length. |
| Approach: | They propose a modified transformer architecture that constrains the encoder-decoder attention mechanism to a subset of input sentences while maintaining system performance. |
| Outcome: | The proposed architecture can be trained and inferenced using large transformer models with expensive training and induction costs. |
Finetuning Pretrained Transformers into RNNs (2021.emnlp-main)
Copied to clipboard
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A. Smith
| Challenge: | Efficient transformers outperform recurrent neural networks in natural language generation, but this comes with significant computational cost and memory footprint during generation. |
| Approach: | They propose to convert a pretrained transformer into its efficient recurrent counterpart, improving efficiency while maintaining accuracy. |
| Outcome: | The proposed transformers outperform recurrent neural networks in natural language generation but come with significant computational and memory footprint during generation. |
RoCEL: Advancing Table Entity Linking through Distinctive Row and Column Contexts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for table entity linking ignore row and column contexts . existing methods for TEL focus on understanding sequential text contexts, making it difficult to adapt to the row and columns structure of tables. |
| Approach: | They propose to leverage row and column contexts to enhance the semantics of mentions in entity disambiguation. |
| Outcome: | The proposed method outperforms the state-of-the-art (SOTA) baseline by 1.5% on the in-domain dataset and 3.7% on average across three out-of domain datasets. |
TellWhisper: Tell Whisper Who Speaks When (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches decouple temporal modeling and speaker modeling when addressing 'when' and 'who' . a new framework that couples temporal structure with speaker dynamics is proposed to address these limitations . |
| Approach: | They propose a framework that couples temporal and speaker identity within the speech encoder . they propose TS-RoPE, a time-speaker rotary positional encoding that partitions Query/Key channels into temporal, speaker subspaces and applies region-specific rotations to align "when" and "who" cues in selfattention. |
| Outcome: | The proposed framework couples temporal structure with speaker dynamics in speech encoder . it uses frame-level speaker activity to estimate speaker-activity estimates . |
On the Expressivity Role of LayerNorm in Transformers’ Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models. |
| Approach: | They propose to use LayerNorm to normalize the activations during the forward pass and their gradients during the backward pass. |
| Outcome: | The proposed model is able to express the multi-head attention layer that follows it in a d-1 space and scales to the same norm of d. |
Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained transformers suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including those unfavorable to classification performance. |
| Approach: | They propose to integrate token pruning and token combining strategies to improve model performance and reduce computational demands. |
| Outcome: | Experiments with various datasets show that the proposed model performs better than baseline models, with the best improvement over the existing model. |
Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using a method to identify next-token neurons, we find that some attention heads recognize contexts relevant to predicting a token and activate a downstream token-predicting neuron accordingly. |
| Approach: | They propose a method to identify next-token neurons and determine the upstream attention heads responsible for their activity in LLMs. |
| Outcome: | The proposed method identifies next-token neurons, finds prompts that highly activate them, and determines the upstream attention heads responsible. |
VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in interpretability suggests we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary space, a transformation that makes them more human interpretable. |
| Approach: | They propose a tool to visualize a forward pass of Generative Pre-trained Transformers as an interactive flow graph with nodes representing neurons or hidden states and edges representing interactions between them. |
| Outcome: | The proposed visualization simplifies huge amounts of data into easy-to-read graphs that can reflect the models’ internal processing, uncovering the contribution of each component to the models' final prediction. |
Medical Entity Disambiguation with Medical Mention Relation and Fine-grained Entity Knowledge (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for medical entity disambiguation (MED) fail to fully utilize the knowledge within medical knowledge bases (KBs) Existing models overlook essential interactions between medical mentions and candidate entities, resulting in knowledge- and interaction-inefficient modeling and suboptimal disambiguations performance. |
| Approach: | They propose to combine a mention relation fusion module and an entity knowledge fusion modules to map medical mentions to corresponding entities in a knowledge base (KB) |
| Outcome: | The proposed method outperforms state-of-the-art MED models on two publicly available real-world datasets. |
MHGRL: An Effective Representation Learning Model for Electronic Health Records (2024.lrec-main)
Copied to clipboard
| Challenge: | Effective EHR representations are key to achieving high performance in healthcare applications. |
| Approach: | They propose a multimodal heterogeneous graph-enhanced representation learning to learn EHR representations using medical ontology and textual notes. |
| Outcome: | The proposed model outperforms baseline models on two real clinical datasets in downstream tasks. |
Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods modify attention mechanism to be bidirectional, undermining LLMs’ ability to extract semantic information acquired during pre-training. |
| Approach: | They propose a general-purpose embedding model that pre-encodes input text into a single Contextual token and then prepends it to the LLM's input sequence. |
| Outcome: | The proposed model improves performance of decoder-only large language models without altering their architectures or introducing significant computational overhead. |
DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metrics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-agent debates can improve the accuracy of Large Language Models by having multiple agents discuss solutions over several rounds of debate. |
| Approach: | a debate framework that uses uncertainty metrics to assess agent confidence is proposed . the framework uses textual prompts or a modified attention mechanism that adjusts token weights . |
| Outcome: | The proposed framework assesses agent confidence using uncertainty metrics . the framework is available at https://github.com/lukeyoffe/debunc. |
MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Decoder-only large language models are increasingly being adapted for bidirectional modeling . however, their reliance on causal attention restricts their effectiveness in tasks that require understanding of bidirectional context. |
| Approach: | They propose a method to adapt decoder-only large language models to generate robust representations and infill missing text spans. |
| Outcome: | The proposed method surpasses strong decoders on token-level and sentence-level representation learning tasks and generates contextually appropriate text infills without excessive repetition of words or phrases. |
StyleFlow: Disentangle Latent Representations via Normalizing Flow for Unsupervised Text Style Transfer (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to separate content from style but some words contain both content and style information. |
| Approach: | They propose a method which uses a reversible encoder to improve content disentanglement. |
| Outcome: | The proposed method outperforms baselines on sentiment transfer and formality transfer tasks. |
SSA: Improving Performance With a Better Scoring Function (2026.acl-long)
Copied to clipboard
| Challenge: | Despite the success of in-context learning, recent studies have identified systematic limitations in its generalization behavior. |
| Approach: | They propose a new attention scoring function that mitigates failures in transformer models . they use Scaled Signed Averaging to train the scoring function instead of Softmax . |
| Outcome: | The proposed scoring function outperforms transformer models with Softmax on NLP benchmarks and linguistic probing tasks. |
Understanding How Positional Encodings Work in Transformer Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have reported superiority of relative PEs in translation tasks. |
| Approach: | They analyze in which part of a transformer model PEs work and compare them using experiments . they find that relative PEs should be added only to query and key of attention mechanism . |
| Outcome: | The results show that relative and absolute PEs work in a transformer model, and should be added to the query and key of an attention mechanism, not to the value. |
NeuRAG: End-to-End Neural Knowledge Augmentation via Hyper-Neurons (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to grounding large language models in external knowledge are constrained by a decoupled architecture: retrieval and reasoning operate as separate stages, with retrieved text merely prepended as passive context. |
| Approach: | They propose an end-to-end Neuralized RAG framework that unifies knowledge retrieval and fusion through Hyper-Neurons. |
| Outcome: | Extensive experiments across multiple datasets and LLMs demonstrate NeuRAG’s strong and consistent performance as a promising novel RAG paradigm. |
Parallel Context-of-Experts Decoding for Retrieval Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Retrieval Augmented Generation relies on concatenating documents into a long context prompt, causing prefill bottlenecks. |
| Approach: | They propose a training-free framework that shifts evidence aggregation from attention to decoding . they treat retrieved documents as isolated "experts", synchronizing their predictions via a retrieval-aware extension of context-awful decoding. |
| Outcome: | The proposed framework shifts evidence aggregation from attention to decoding . it treats retrieved documents as isolated experts, synchronizing their predictions . |
DrFrattn: Directly Learn Adaptive Policy from Attention for Simultaneous Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn read/write policies from attention mechanism may compromise effectiveness of attention mechanism . |
| Approach: | They propose a method that directly learns adaptive policies from the attention mechanism . experimental results demonstrate that the method achieves an improved balance between translation accuracy and latency. |
| Outcome: | The proposed method achieves improved balance between translation accuracy and latency. |
A Syntactic and Semantic Probe into Language Evolution based on Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on language evolution have relied on manual annotated resources and rely on dependency parsing. |
| Approach: | They propose to use attention-based structural distance and semantic space distance to measure language development. |
| Outcome: | The proposed measures show that human and LLMs share common characteristics in language processing. |